Skip to content

zephyr: ll-schedule: fix a race between task cancel and new schedule - #4898

Merged
lgirdwood merged 1 commit into
thesofproject:mainfrom
kv2019i:topic/zpauseresume-fix
Oct 25, 2021
Merged

lgirdwood merged 1 commit into
thesofproject:mainfrom
kv2019i:topic/zpauseresume-fix

Conversation

@kv2019i

@kv2019i kv2019i commented Oct 20, 2021

Copy link
Copy Markdown
Collaborator

As task cancellation is an asynchronous operation, it is possible that
the task is in SOF_TASK_STATE_CANCEL state when zephyr_ll_task_schedule()
is called. With current code, the cancel takes precedence and task is
not scheduled again.

This shows up as errors in SOF audio test case for multiple-pause-resume
where a set of pipelines is repeatedly paused and unpaused. If the race
is hit, a request to pause is successful (and pipeline task is requested
to be cancelled) but the subsequent release will not get schedueld at all
leading to an IPC timeout.

BugLink: #4742
Signed-off-by: Kai Vehmanen kai.vehmanen@linux.intel.com

@kv2019i
kv2019i requested a review from lyakh October 20, 2021 13:27
@kv2019i
kv2019i requested a review from mrajwa as a code owner October 20, 2021 13:27
@kv2019i
kv2019i requested review from lgirdwood and marc-hb October 20, 2021 13:42

@lyakh lyakh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch! What is happening here is that when a task is cancelled, the zephyr-ll scheduler only changes its status to cancelled and expects the next scheduler run to actually remove the task from the list. However, if instead the stream is unpaused before the next scheduler cycle gets a chance to run, then we come to a wrong conclusion, that the task is already scheduled, missing the fact that its status is "cancelled," so we don't schedule it again.
The fix is good, it can be done a bit differently but both versions are fine with me.

Comment thread src/schedule/zephyr_ll.c Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alternatively we could put this test above line 316 inverted and avoid running the loop completely, but maybe this way it's a bit more obvious. Although this does come with a bit of performance penalty compared to the other version :-)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lyakh hmm, but we can't know for sure "task" is already on the "sch->tasks" list. So I'd keep it here.

I uploaded a new version now. I realized we also need to avoid re-adding the task to the "sch->tasks" lists.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if it is not on the list, we anyway continue after the loop, right? Same as if it is on the list and cancelled. So, the list only makes sense for not cancelled tasks on it.

@kv2019i

kv2019i commented Oct 20, 2021 •

Copy link
Copy Markdown
Collaborator Author

Hmm, Intel internal test plan #7544 is still showing error with (UPDATE) V1 patch. I have two local TGL machines and with both, without this patch, multiple-pause-resume fails with enough iterations. But in the test plan (TGL machine with nocodec tplg), there is still a failure. So it is possible more fixes are needed still. This fix seems to be still valid.

@kv2019i

kv2019i commented Oct 21, 2021

Copy link
Copy Markdown
Collaborator Author

@lyakh Hmm, let's hold off one this. I started to worry whether the cancelled entry in the task list is cleaned right after this patch.

As task cancellation is an asynchronous operation, it is possible that
the task is in SOF_TASK_STATE_CANCEL state when zephyr_ll_task_schedule()
is called. With current code, the cancel takes precedence and task is
not scheduled again.

This shows up as errors in SOF audio test case for multiple-pause-resume
where a set of pipelines is repeatedly paused and unpaused. If the race
is hit, a request to pause is successful (and pipeline task is requested
to be cancelled) but the subsequent release will not get schedueld at all
leading to an IPC timeout.

BugLink: thesofproject#4742
Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
@kv2019i
kv2019i force-pushed the topic/zpauseresume-fix branch from 0e102bc to 7a6cd1d Compare October 21, 2021 14:30
@kv2019i
kv2019i marked this pull request as ready for review October 21, 2021 16:47
@kv2019i
kv2019i requested a review from lyakh October 21, 2021 16:47
@kv2019i

kv2019i commented Oct 21, 2021

Copy link
Copy Markdown
Collaborator Author

Pushed a new version. This addresses a problem of adding the same task multiple times in the task list. As all instances point to same task struct, they won't be cleaned up as expected. This should be harmless, but for some strange reason with this V2 patch, the multiple-pause-resume test passes also with TGL nocodec topology.

@lyakh

lyakh commented Oct 22, 2021

Copy link
Copy Markdown
Collaborator

Pushed a new version. This addresses a problem of adding the same task multiple times in the task list. As all instances point to same task struct, they won't be cleaned up as expected. This should be harmless, but for some strange reason with this V2 patch, the multiple-pause-resume test passes also with TGL nocodec topology.

Yes, the new version makes sense! Not sure to what part of your explanation "harmless" is referring, but adding the same element to a list multiple times without deleting it is definitely anything but harmless!

@lgirdwood

Copy link
Copy Markdown
Member

CI showing some not tested DUTs and an unrelated issue on BYT.

@lgirdwood
lgirdwood merged commit b1ee4a2 into thesofproject:main Oct 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants